On the use of the multimodal clues in observed human behavior for the modeling of agent cooperative behavior

We introduce TYCOON a framework we are developing for the analysis of human verbal and non-verbal behavior. This framework includes a typology made of six primitive types of cooperation between communicative modalities: equivalence, specialization, transfer, redundancy, complementarity and concurrency. We have used this typology when annotating videotaped multimodal humancomputer interaction and human-human communication. We have defined some metrics for the analysis of such multimodal behavior of observed subjects. The computed values of such metrics are being used for the specification of the multimodal behavior of an embodied conversational agent. We will present how these low-level specifications of combination between modalities might be used in the future as building blocks for bridging the gap between experimental human behavior analysis and the modeling of cooperation-related multimodal attitudes related for instance to autonomy, delegation and control.